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(57) ABSTRACT 

A pixel filling system or method begins with an initial set of 
image data that is a subset of data values in a data array i ki 
representing data values at corresponding positions (k,l). 
Some of the f kJ values are initially undefined, while others, 
at positions where both k and 1 are even numbers, have 
initially defined values. An interpolated value is generated 
for initially undefined f* , values in accordance with the pixel 
filling filter equation: 



wherein n is a positive integer, d iV are the subset of Rvalues 
which have defined values prior to the generation of the 
interpolated i kl values; V kl are coefficients, no more than 
sixteen of which have non-zero values; and each non-zero 
? kl coefficient is an integer divided by an integer power of 
two. The pixel values generated in accordance with the pixel 
filling filter equation represent a surface that is smooth, has 
continuous two-dimensional spatial first derivative, and pre- 
serves sharp edges, if any, in the initial set of image data. 
Further, if a local set of the initial data points d fj - (e.g., 
neighboring ones of the ^2,7/2 or U/4,i/4 points) all fall on a 
polynomial surface, the interpolated i k j values that are 
located between the initial data points also fall on that same 
polynomial surface. 

37 Claims, 13 Drawing Sheets 
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IMAGE DATA INTERPOLATION SYSTEM 
AND METHOD 

The present invention relates generally to "filling in" the 
RGB data obtained from digital camera sensors, and to 5 
generating interpolated image data values when a digitally 
encoded image is magnified to a larger size, and particularly 
to a system and method for performing image data interpo- 
lations using spatially limited two-dimension "FIR-like" 
filters and yet generating images that appear continuous, io 
undistorted and sharp. 

BACKGROUND OF THE INVENTION 

The present invention addresses two related problems: 
filling in RGB data obtained from digital camera sensors, 15 
and to generating interpolated image data values when a 
digitally encoded image is magnified to a larger size. 

Referring to FIG. 1, the first problem concerns the fact 
that digital camera sensors generate only one color value 2Q 
(for red, green or blue) at each pixel position of the image, 
creating a pattern of red, green and blue pixels called a 
mosaic. The specific pattern shown in FIG. 2 is referred to 
as the Bayer mosaic pattern. However, all three R, G, B color 
values are needed for every pixel position to produce a 2S 
full-color image. In this situation, when image data is first 
received from the camera sensor, only half the pixels in the 
image contain an intensity value for the color green, only 
one fourth of the pixels in the image contain an intensity 
value for the color blue, and only one fourth of the pixels in 3Q 
the image contain an intensity value for the color red. The 
goal is to produce the most "accurate" possible intensity 
values of the missing green, blue and red intensity values, 
essentially predicting what those intensity values would 
have been if the camera sensor had been able to record them. 35 

Referring to FIG. 2, the second problem concerns the 
magnification of an image, such as by a factor of four (i.e., 
the width and height of the image are each doubled). In this 
situation, it is assumed that the system already has RGB or 
YUV values, or some other equivalent set of values, repre- 40 
senting the initial set of image pixels. However, three fourths 
of the pixels in the magnified image initially have no image 
data values, and thus those values have to be generated from 
the information available in the original pixels (I), which 
have now been mapped to all the even index positions in the 45 
magnified image (J). For magnification factors other than 
four, the number of pixels that initially have no value will 
depend on the value of the magnification factor. 

While there are literally hundreds of articles and patents 
addressing interpolation techniques for handling the missing 50 
RGB data problem and image data magnification, there are 
three primary image data interpolation techniques that are 
used in prior art systems: (1) pixel replication, (2) linear/ 
bilinear interpolation, and (3) cubic/bicubic interpolation. 
Pixel replication is the simplest image data interpolation 55 
technique. In pixel replication, each interpolated pixel value 
is generated simply by replicating the value of a neighboring 
pixel in the initial set of pixels. The main advantage of pixel 
replication is that the interpolation is computationally very 
simple. The primary disadvantage of this technique is that eo 
the quality of the resulting image is poor. 

In linear/bilinear interpolation, each interpolated pixel 
value is generated by essentially "averaging" the values of 
the two or four closest neighboring pixels in the initial set of 
pixels. Linear interpolation is used when performing data 65 
interpolation on a one-dimensional data set. Linear interpo- 
lation generates the interpolated pixel value based on the 



"average" of the values of the two closest neighboring 
pixels, one on the left and one on the right, and is a function 
having terms proportional to 1 and x, where x is the 
coordinate of the interpolated pixel value. Bilinear interpo- 
lation is used when performing image data interpolation on 
a two-dimensional image. Bilinear interpolation generates 
the interpolated pixel value based on the "average" of the 
four closest neighboring pixels, one in each of the northwest, 
northeast, southeast and southwest directions, and is a 
function having terms proportional to 1, x, y and xy, where 
x and y are the coordinates of the interpolated pixel value. 
Although the quality of the resulting image is better than that 
produced by pixel replication, the image quality is still 
relatively poor. 

In cubic/bicubic interpolation, each interpolated pixel 
value is generated based on the values of all pixels in the 
initial set of pixels. Each interpolated pixel is set to a value 
such that the piecewise cubic polynomial (sometimes called 
a cubic spline) formed by the pixels in the resulting image 
has continuous first and second derivatives (i.e., has second 
order smoothness). Cubic interpolation is used when per- 
forming data interpolation on a one-dimensional data set. 
The interpolated pixel value generated using cubic interpo- 
lation is a function having 1, x, x 2 and x 3 terms, where x is 
the coordinate of the interpolated pixel value. Bicubic inter- 
polation is used when performing image data interpolation 
on a two-dimensional image. The interpolated pixel value 
generated using bicubic interpolation is a function having 1, 
x, y, xy, x 2 , x 2 y, x 2 )^, x^ 3 , x 3 , x 3 y, x 3 y 2 and x 3 y 3 terms, 
where x and y are the coordinates of the interpolated pixel 
value. The quality of the image produced by cubic/bicubic 
interpolation is usually very good. Unfortunately, however, 
the computation and memory requirements of cubic/bicubic 
interpolation is very high because all pixel values in the 
image are required to generate each interpolated pixel value. 

Goals of Present Invention 

Goal 1: Low Distortion. Alternately stated, it is a goal to 
avoid the use of higher order cross terms, herein called 
"distortion" terms because they are responsible for 
introducing artifacts and distortions in interpolated 
image data. For bilinear interpolation, the distortion 
term is xy, having a "total power" of two. For biqua- 
dratic interpolation, the distortion terms are terms hav- 
ing a "total power" greater than two (namely x 2 y, xy 2 
and x 2 )^). For bicubic interpolation, the distortion 
terms are those having total power greater than three 
(namely x 3 y, x 3 y 2 , x 2 y 2 , x^ 3 , x 2 ^ and xy 3 ) Distortion 
terms are undesirable because they increase the 
memory required to generate the interpolated image 
data and generate image artifacts along the diagonals of 
each rectangular region formed by any four adjacent 
pixels. 

Goal 2: Smoothness. The interpolated image data should 
represent a smooth and continuous image surface, 
without discontinuities. If a group of original, neigh- 
boring image data points lie on a polynomial surface 
(e.g., plane or quadratic surface), the new interpolated 
image data points used to fill in holes between the 
original image data points should lie on the same 
polynomial surface. 

Goal 3: Continuous turning tangent planes. The tangent 
planes of the image surface represented by the inter- 
polated image data should be "continuously turning," 
without discontinuities. In other words, the two- 
dimensional spatial first derivative of the surface 
should be continuous, providing a surface without any 
sharp angles. 
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Goal 4: The tangent plane of the image surface repre- present invention. In the second stage the pixel values 

seated by the interpolated image data at each location resulting from the first stage are remapped to new locations 

for which an interpolated value is generated should be based on the specified magnification factor. This mapping 

substantially parallel (i.e., visually indistinguishable will usually map some pixel values to non-pixel positions, 

from a plane that is parallel) to a secant plane formed 5 and will always leave many pixel positions with no assigned 

by adjacent ones of the data points corresponding to the P ixel value. Anew type of filter, herein called a "Barycentric 

initial data values interpolation filter," is used to compute pixel values at each 

• . ■ • of the pixel positions P not assigned a pixel value by the 

Goals 12 and 13 ensure that the image .after mterpolation \ £ or eacn such ixe t position P> a refer e nce 

is "pleasing by minim izmg the visibility of artifacts in tdangle ^ by ^beting the three closest pixel 

the interpolated image such as lines. Goal 4 ensures that io values> a , posilions A B and c> having pixel values a> b and 

sharp lines m the image remain sharp or near sharp in Cj respec tively. The coordinates of those pixel values and the 

the interpolated image. p j xe j pQ sition P for which a pixel value p is to be generated 

SUMMARY OF THE INVENTION are ™ a PP ed / nto lf ! e Barycentric coordinate system. The 

coordinates (u, v, w) of the pixel position P in the Barycen- 

In summary, the present invention is a system and method 35 trie coordinate system are then used as weights for gener- 

for generating interpolated pixel data values that begins with ating a pixel value p=ua+vb+wc that lies on the plane that 

an initial image data set that is a subset of data values in a passes through the vertices A, B and C of the reference 

data array i k j representing data values at corresponding triangle, 

positions (k,l). Some of the f* , values arc initially undefined, BRIEF DESCRIPTION OF THE DRAWINGS 

while others at positions where both k and 1 are even 20 AJJ ... i . ^ c * c *u • n u 

. , • -f- ii j n j i Additional objects and features of the invention will be 

numbers have initially defined values. mQre feadily ^ ^ mo „ ing detailed descrip . 

For filling in midpoints (half way between pixels having tion and appende d claims when taken in conjunction with 

assigned values), an interpolated value is generated for each tne drawings, in which: 

initially undefined i kl value in accordance with the pixel p IG j repreS ents the grid of R, G and B data values 

filling filter equation: generated by a typical digital camera sensor. 

FIG. 2 represents the original and "fill in" pixels of an 

Ik i = V Pk-2ij-2j<i< j (1) image that has been magnified by a factor of four. 

ij J J FIG. 3 is a block diagram of a digital camera incorporat- 

3Q ing an embodiment of the present invention. 

FIG. 4 is a block diagram of a general-purpose computer 

wherein incorporating an embodiment of the present invention. 

d, v are the subset of t kJ values which have defined values FIG. 5 depicts a pixel filling filter in accordance with the 

prior to the generation of the interpolated i kl values; present invention. 

Pjt , are coefficients, no more than sixteen of which have 35 FIGS. 6A-6F depict the non-zero coefficients and the 

non-zero values; and each non-zero ? k l coefficient is an pixel data values to which they are applied for various 

integer divided by an integer power of two. midpoint pixel filling filters. FIG. 6Apertains to a horizontal 

For filling in quarter points, a filter equation analogous to midpoint pixel filling filter. FIG. 6B pertains to a vertical 

equation 1, but with 2i and 2j replaced by 4i and 4j is used. midpoint pixel filling filter. FIG. 6C pertains to a two 

Alternately, filtering using equation is performed twice, but 40 -dimensional center midpoint pixel filling filter. FIGS. 6D 

the resulting image will usually be less sharp, and 6E represent edge-condition midpoint pixel filling filters 

The pixel values generated in accordance with the pixel for horizontal and center midpoints, and FIG. 6F pertains to 

filling filter equation have the following properties: a comer-condition center midpoint pixel filling filter. 

(A) the set of Rvalues (including the mapped d„ values) FIGS - 7A and 7B de P ict ^° embodiments of a pixel 
represent a smooth surface that is continuous^ when- 45 fiUin & fiIter for com P utin & fill - in P ixel values for an i™ge 
ever a local set of the initial data points d, , (i.e., that has beeD magnified by a factor of sixteen (i.e., by a 
neighboring ones of the f*, points where both k and 1 factor of four in each dimension). 

are even) all fall on a polynomial surface (e.g., plane or F1GS - 8A-8G depict the non-zero coefficients and the 

paraboloid), the interpolated f kt values that are located pixel data values to which they are applied for two-level, 

between the initial data points also fall on that same 50 quarter-point pixel filling filters. FIGS. 8A and 8B pertain to 

polynomial surface; two different horizontal quarter-point pixel filling filters. 

(B) the surface represented by the set of f w values has a FIGS - 8C and 80 P ertain t0 two different vertical 1 uartcr - 
two-dimensional spatial first derivative that is continu- P oim P ixel fimn S filters - F1GS - 8E ' 8F ™ 6 80 P ertain 10 
ous- and mree different two-dimensional center quarter-point pixel 

(C) the surface represented by the set of f„ values has at 55 fiUin S filte ^ for ,hree different classes of center ( > uarter - 

each location (k,l) for which an interpolated value is P oint P^ 6 ^- 

generated a tangent plane that is parallel or substan- FIG * 9 is a flow chart d epicting a method of generating 

tially parallel to a secant plane formed by adjacent ones P^ 1 data values when an ima 8 e 15 magnified by a factor that 

of the data points (k,l) corresponding to the initial data 15 no1 e< l ual to an mle S er P ower of four - 

values d . 60 F ^ G - 10 depicts the pixels generated by the method of 

To magnify'an image by a specified factor other than an F ' G - 9 when an image is magnified by a factor of nine, 

integer power of four, the image is magnified in two stages. DESCRIPTION OF TOE PREFERRED 

In the first stage the image is magnified by a factor that is EMBODIMENTS 
smaller than the specified factor, but which is an integer 

power of four. (The first stage is skipped if the specified 65 Dl S llal Camera Embodiment 

magnification factor is less than four.) The newly created Referring to FIG. 3, there is shown an embodiment of a 

pixels are generated using the pixel filling filter of the digital camera system 100 in accordance with the present 
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invention. The digital camera system 100 includes an image 
capture device 102, such as a CCD sensor array or any other 
mechanism suitable for capturing an image as an array of 
digitally encoded information. Thus the image capture 
device is assumed to include analog to digital conversion 5 
(ADC) circuitry for converting analog image information 
into digital values. 

A working memory 104, typically random access 
memory, receives digitally encoded image information from 
the image capture device 102. More generally, it is used to 10 
store a digitally encoded image while the image is being 
transformed and compressed and otherwise processed by the 
camera's data (i.e., image) processing circuitry 106. The 
data processing circuitry 106 in one embodiment consists of 
hardwired logic and a set of state machines for performing 15 
a set of predefined image processing operations. 

In alternate embodiments the data processing circuitry 
106 could be implemented in part or entirely using a fast 
general-purpose microprocessor and a set of software pro- 
cedures. However, at least using the technology available in 
1998, it would be difficult to process and store mil resolution 
images (e.g., full color images having 1280x840 pixels) fast 
enough to enable the camera to be able to take, say, 20 
pictures per second, which is a requirement for some com- 
mercial products. If, through the use of parallel processing 
techniques or well designed software, a low voltage general- 
purpose image data microprocessor could support the fast 
image processing needed by digital cameras, then the data 
processing circuit 106 could be implemented using such a 
general-purpose microprocessor. 

Each image, after having been processed by the data 
processing circuitry 106, is typically stored as an "image 
file" in a nonvolatile memory storage device 108, typically 
implemented using "flash" (i.e., EEPROM) memory tech- 3S 
nology. The nonvolatile memory storage device 108 is 
preferably implemented as a removable memory card. This 
allows the camera's user to remove one memory card, plug 
in another, and then take additional pictures. However, in 
some implementations, the nonvolatile memory storage 4Q 
device 108 may not be removable, in which case the camera 
will typically have a data access port 110 to enable the 
camera to transfer image files to and from other devices, 
such as general-purpose, desktop computers. Digital cam- 
eras with removable nonvolatile memory 108 may also 45 
include a data access port 10. 

The digital camera 100 includes a set of buttons 112 for 
giving commands to the camera. In addition to the image 
capture button, there will typically be several other buttons 
to enable the use to select the quality level of the next picture 50 
to be taken, to scroll through the images in memory for 
viewing on the camera's image viewer 114, to delete images 
from the nonvolatile image memory 108, and to invoke all 
the camera's other functions. Such other functions might 
include enabling the use of a flash light source, and trans- 55 
ferring image files to and from a computer. The buttons in 
one embodiment arc electromechanical contact switches, but 
in other embodiments at least some of the buttons may be 
implemented as touch screen buttons on a user interface 
display 116, or on the image viewer 114. 60 

The user interface display 116 is typically implemented 
either (A) as an LCD display device separate from the image 
viewer 114, or (B) as images displayed on the image viewer 
114. Menus, user prompts, and information about the images 
stored in the nonvolatile image memory 108 may be dis- 65 
played on the user interface display 116, regardless of how 
that display is implemented. 



After an image has been captured, processed and stored in 
nonvolatile image memory 108, the associated image file 
may be retrieved from the memory 108 for viewing on the 
image viewer. More specifically, the image file is converted 
from its transformed, compressed form back into a data 
array suitable for storage in a framebuffer 118. The image 
data in the framebuffer is displayed on the image viewer 114. 

The digital camera 100 preferably includes data process- 
ing circuitry 106 for performing a predefined set of primitive 
operations, such as performing the multiply and addition 
operations required to apply an "FIR -like" (finite impulse 
response-like) filter to a certain amount of image data, as 
well as a set of state machines 130-138 for controlling the 
data processing circuitry so as to perform a set of predefined 
image handling operations. In one embodiment, the state 
machines in the digital camera are as follows: 

One or more state machines 130 for transforming, com- 
pressing and storing an image received from the cam- 
era's image capture mechanism. This image is some- 
times called the "viewfinder" image, since the image 
being processed is generally the one seen on the 
camera's image viewer 114. This set of state machines 
130 are the ones that initially generate each image file 
stored in the nonvolatile image memory 108. Prior to 
taking the picture, the user specifies the quality level of 
the image to be stored, using the camera's buttons 112. 
With respect to the present invention, this state machine 
controls the process of generating the "missing" RGB 
values not generated by the image capture device 102. 
One or more state machines 132 for decompressing, 
inverse transforming and displaying a stored image file 
on the camera's image viewer. The reconstructed image 
generated by decompressing and inverse transforming 
the image data is stored in camera's frame buffer 118 so 
that it can be viewed on the image viewer 114. 
One or more state machines 134 for implementing a 
"viewfinder 1 ' mode for the camera in which the image 
currently "seen" by the image capture mechanism 102 
is displayed on the image viewer 114 to that the user 
can see the image that would be stored if the image 
capture button is pressed. These state machines transfer 
the image received from the image capture device 102, 
possibly after appropriate remedial processing steps are 
performed to improve the raw image data, to the 
camera's framebuffer 118, These state machines 134 
also implement the image magnification portion of the 
present invention. 
One or more state machines 136 for downloading images 
from the nonvolatile image memory 108 to an external 
device, such as a general-purpose computer. 
One or more state machines 138 for uploading images 
from an external device, such as a general-purpose 
computer, into the nonvolatile image memory 108. This 
enables the camera to be used as an image viewing 
device, and also as a mechanism for transferring image 
files on memory cards. 
The image capture and viewfinder state machines 130, 
134 are preferably arranged in a pipeline so that RGB 
interpolation by the image capture state machine 130 and 
image enlargement by the viewfinder state machine 134 can 
be done in a pipeline, with image enlargement being per- 
formed on portions of the image without waiting for the 
entire image to be stored in memory. In this way, an image 
can be RGB interpolated and enlarged in about the same 
amount of processing time it would take to perform either of 
those operations alone. 
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General-Purpose Computer Implementation 

Referring to FIG. 4, the present invention may also be 
implemented using a programmed general-purpose com- 
puter system 150. The computer system 150 includes: 
one or more data processing units (CPU's) 152, 
memory 154, which will typically include both high speed 
random access memory as well as non-volatile 
memory; 
a user interface 156; 

a network or other communication interface 158 for 
communicating with other computers as well as other 
devices; 

a data port 160, such as for sending and receiving images 
to and from a digital camera (although such image 
transfers might also be accomplished via the network 
interface 158); and 

one or more communication busses 162 for interconnect- 
ing the CPU(s) 152, memory 154, user interface 156, 
network interface 158 and data port 160. 

The computer system's memory 154 stores procedures 
and data, typically including: 

an operating system 172 for providing basic system 
services; 

a file system 174, which may be part of the operating 
system; 

application programs 176, such as user level programs for 

viewing and manipulating images, 
an image processing module 178, for performing various 

image processing functions including those that are the 

subject of the present document; 
image files 180 representing various images. 
The image processing module 178 may include: 
an RGB interpolator 182 for converting a raw image file, 

in which each pixel only has a red, green or blue 

intensity value, into a full RGB image data file; and 
an image magnifier module 190. 

The RGB interpolator 182, in turn, may include a set of 
midpoint pixel filling filters 184. 

The image magnifier module 190, in turn, may include: 

a set of midpoint pixel filling filters 192; 

a set of quarter-point pixel filling filters 194; and 

a barycentric interpolator procedure 196. 

In a preferred embodiment, the RGB interpolator 182 and 
the image magnifier module 190 are coupled in a pipeline - 
like manner so that RGB interpolation by the RGB interpo- 
lator 182 and image enlargement by the image magnifier 
module 190 are performed in overlapping time periods, with 
image enlargement being performed on portions of the 
image without waiting for the entire image to be first 
processed by the RGB interpolator 182. In this way, the 
system can display some portions of an enlarged image on 
the system's display device while it still processing (RGB 
interpolating and enlarging) other portions. 

Filter Interpolation 

The present invention achieves high quality image 
magnification, as well as RGB data value interpolation, 
while avoiding the computational complexity and extensive 
memory requirements of the bicubic interpolation technique. 
In particular, the present A invention uses "short" (i.e., 
spatially limited) two-dimensional finite impulse response 
(FIR)-like filters to generate interpolated image, data values. 
In virtually all cases, the number of "tap points" in the filter 
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having non-zero coefficients is less than sixteen, and for 
midpoint interpolation values (as opposed to quarter-point 
interpolation values) the number of non-zero coefficients is 
in all cases is eighteen or less. In practice, the number of 

5 non-zero coefficients in most useful interpolation filters is 
generally twelve or less. Thus, compared to bicubic 
interpolation, which uses all data values in the entire image 
to generate, the present invention is very simple computa- 
tionally. Furthermore, every coefficient of the pixel filling 

10 filters used in the present invention is equal to either an 
integer or an integer divided by an integer power of two. 
Thus, the interpolation values generated by the present 
invention can all be computed using bit shifters and adders. 
Multipliers, and especially floating point multipliers, are not 

is needed. 

Referring to FIG. 5, the basic technique for magnifying an 
image, or a portion of an image, is to "up-sample" the data 
and then apply a convolution filter to the up-sampled data. 
The combined operations are herein called a pixel filling 

20 filter (PFF). The up sampling step generates additional data 
points, usually called pixels. For a two-dimensional image 
magnified by a factor of four (i.e., by a factor of two in each 
dimension), up-sampling doubles the number of pixels in 
each spatial dimension and increases the total number of 

25 pixels by a factor of four. The input to the up-sampling filter 
are a two-dimensional set of data values herein labeled d tJ -, 
the output of the up sampling filter is a set of data values 
labeled 6 k l ~ and the output of the PFF is a set of data values 
labeled f k V 

30 For odd values of k and/or odd values of 1, d~ k l is equal 
to 0, and for even values of both k and 1, 

f k t = dZ, = d k t for both k,l even ^ 
' *' 2'2 

35 

In other words, the original pixel data values at positions 
i,j are moved to positions 2i,2j, and all the "new" pixel data 
values are initially set to zero. Restated, f 2lj 2,-=d l j for integer 
40 values of i and j that fall within the initial set of image data. 

The PFF operates in accordance with the following filter 
formula: 

45 /u = X>*-2;,,-:A, (3J 



where P*.^/-^ are the coefficients of the filter. As indicated 
above, only a very small number of the filter coefficients 

50 Pjt_2,;/_2/ nave non-zero values, and further the positions of 
all the non-zero coefficients are relatively close to the fill-in 
pixel i kl . Furthermore, in the preferred embodiments, the 
sum of the non-zero coefficients used in Equation 3 above is 
always equal to one. Generally, when both k and 1 are even 

55 numbered values, such as 0,0 or 2,4, the only non-zero 
coefficient in Equation 3 above will be the P 0 0 coefficient, 
which is always equal to 1 . When either k or 1 or both are odd 
numbered values, the P 0 0 coefficient is not used in Equation 
3, because either k— 2i or l-2j cannot be equal to zero. 

60 In this document, the coefficients for the various Pixel 
Filling Filters will be shown in two forms: (A) a figure 
showing the non-zero coefficients and pixel data values to 
which they are applied, relative to the fill-in pixel data value 
being generated, and (B) an equation or matrix representa- 

65 tion identifying the values of the non-zero coefficients. In 
both cases, the non-zero coefficients are used in pixel filling 
filter Equation 3 to generate interpolated pixel fill-in values. 
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Every one of the interpolation filters described below 
provides the following properties: 

(A) Smoothness: the set of interpolated i kl values 
(including the mapped d, v values) represent a smooth 
surface that is continuous; if a local set of the initial 
data points d iV (i.e., neighboring ones of the i k t points 
where both k and 1 are even) all fall on a polynomial 
surface (e.g., plane or paraboloid), the interpolated i kJ 
values that are located between the initial data points 
also fall on that same polynomial surface; 

(B) Continuous turning tangent planes: the surface rep- 
resented by the set of interpolated i kl values has a 
two-dimensional spatial first derivative that is continu- 
ous; and 

(C) Sharp edges: the surface represented by the set of 
interpolated i kl values in conjunction with the de- 
values has at each location (k,l) for which an interpo- 
lated value is generated a tangent plane that is parallel 
or substantially parallel to a plane formed by adjacent 
ones of the data points (k,l) corresponding to data 
values d tJ -. 

The phrase "a tangent plane substantially parallel to a 
second plane" is defined, for the purposes of this document 
to mean that the tangent plane is visually indistinguishable 
from a plane that is truly parallel to the second plane within 
the relevant area of the image (e.g., within a region of ten 
pixels in any direction from the interpolation point at which 
the tangent plane is defined). When the tangent plane at each 
interpolation point is substantially parallel to the secant 
plane defined by adjacent data points, sharp edges in the 
image data are preserved. For instance, if there is a sharp 
change in the color red between one region of an image and 
another region of the image (e.g., in an image of a red box 
sitting on a brown table), then the gradient of the red pixel 
values at each interpolation point between the two regions 
should be substantially equal to the gradient defined by the 
original data points. 

Midpoint Pixel Filling Filters 

The midpoint pixel filling filters are used to generate pixel 
fill-in values that are located exactly in the middle of two (or 
more) pixels that have assigned data values. Later in this 
document various "quarter-point" PFF's will be presented. 

Horizontal and Vertical One-Dimensional Pixel 
Filling Filters 

Referring to FIGS. 6A and 6B, the simplest of all the 
PFF's are the horizontal and vertical midpoint pixel filling 
filters. In FIG. 6A, the filter coefficient values are written 
next to the data points to which they would be applied (i.e., 
whose values they would be multiplied by). Using the 
convention that the fill-in pixel is located at position 0,0, the 
non-zero coefficient values for the horizontal midpoint PFF 
are: 



20 



25 



30 



35 



40 



,9, 9, -1) 



(4) 



50 



All other coefficient values for the horizontal midpoint 
PFF are equal to zero. For every fill-in pixel value shown in 
FIG. 6 A, except those closest to the edges of the image array, 
the fill-in value is determined by summing the four closest 
data points multiplied by the coefficients shown in FIG. 6A. 

It should be noted that the pixels at locations having even 
numbered indices, such as f 00 and f 0 2 and the like, are 
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assigned original image data values, and thus for these pixel 
locations the only non-zero PFF coefficient is P 00 , which is 
equal to 1. 

FIG. 6B shows the PFF coefficients for a vertical midpoint 
pixel filling filter. The non-zero coefficient values for the 
vertical midpoint PFF are: 



^0^)= ye 1 " 1 ' 9 ' 9 '~ 1J 



(5) 
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All other coefficient values for the vertical midpoint PFF 
are equal to zero. 

One practical application of the vertical midpoint PFF is 
for filling in rows (lines) of values between the horizontal 
scan lines of an interlaced video image. Each frame of such 
video images contain only half as many horizontal lines of 
pixel data as a full resolution image. In other words, the 
initial set of image data includes pixel values for alternating 
horizontal rows of the data array, and the other rows of the 
array initially do not have assigned pixel values. Using the 
vertical midpoint PFF of the present invention, each row 
(line) of missing pixel data is generated by filtering (in 
accordance with equation 5) the two lines of original data 
above and the two lines of original data below the line of 
data to be generated. The topmost and bottom missing lines 
of the image can be generated using simple averaging, or 
pixel replication, or even better, the non-symmetric edge- 
condition horizontal midpoint PFF of equation 9, discussed 
below. 

TWo-Dimensional "Center" Midpoint Pixel Filling 
Filters 

FIG. 6C shows the PFF coefficients for a two-dimensional 
"center'' midpoint pixel filling filter. This is the filter used to 
generate a data value for a pixel that is located at the center 
position between four data points, but is not at the same x or 
y position as any of those four data points. Using the 
convention that the fill-in pixel is located at position 0,0, the 
non-zero coefficient values in Equation 3 for the center 
midpoint PFF are: 



45 



P-3.L, 

P-x-u 



P-l* 
P-l.U 
P-L-i, 
P-1.-3, 



Pi* 

Pui* 

PI.-I. 



P» 
^3.-3 



-1 0 

8 -1 

2 0 

0 0 



(6) 



All other coefficient values for the center midpoint PFF 
are equal to zero. It is noted that the sum of the non-zero 
coefficients for this PFF, as well as all other PFF's in the 
preferred embodiment, is equal to one. 

Other embodiments of the center midpoint PFF include 
55 one with the same coefficients, except reversed with respect 
to either the X or Y axis (either reversal produces the exact 
same alternate PFF): 



60 



P-W P-13* Pl.3* Pii 

P-1,1, P-i.u Pi.u Pxi 

P-3,-1* P-l-U Pl,-l* P\-l 

P-3,-3, P-1.-3. Pi, -3. ^3,-3 



0-100 
-18 2 0 
0 2 8 -1 
0 0-10 



(7) 



65 Reversing the filter with respect to both the X and Y axes 
reproduces the original filter. Another embodiment of the 
center midpoint PFF includes one that is the average of the 
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PFF filter described above and the one where the coefficients 
are reversed with respect to either axis. The coefficients for 
that embodiment are: 



P- 3 .i< P-I.U P\.\* ^3.1 
P —3,-1 • P-l.-l. Pl.-l. Pi,~l 
P-1.-3, /'l.-J. ^3.-3 



( 0 -1-1 0 

-1 10 10 -1 

-1 10 10 -1 
0-1-10 



(P_ 3 ,o, P_ 1>( >, P1.0, P3.0) = jg(-2, 12, 6, 0) 



(9) 



FIG. 6E depicts a pixel filling filter for a center midpoint 
that is near the left edge of the image data array. The 
non-zero coefficient values for the center, near-edge mid- 
point PFF are: 



( P_3,3. P-IJ, ^1.3, ^3,3 

P_ 3>1 , P-l.l, P M , P 3l l 

P-3,-1, P-l.-l, P3.-I 

P-3,-3. P-1,-3, ^1.-3, ^3.-3 



( 0 0-10 
0 3 6 0 
-2' 10 1 0 
0-100 





P-l^ 


Pl.3. 


^3.3 1 




' 0 


0 


0 


0 1 


P-3.1 . 


P-IA* 


Pl.l. 


^3.1 


1 


0 


2 


4 


0 


P-3.-I. 




Pl.-I. 




~ 16 


-2 


12 


2 


0 






P|,-3, 


Py-3 j 




, 0 


-2 


0 


0 ; 



40 



(10) 



FIG. 6F depicts a pixel filling filter for a center midpoint 
that is near a comer of the image data array. The non-zero 
coefficient values for the center, near-corner midpoint PFF 50 
are: 



(ID 



The above-described midpoint PFF's are used to generate 
an RGB image from image data provided by a digital camera 60 
sensor. Referring to FIGS. 1 and 6A-6F, the red, green and 
blue fill-in pixel values of the RGB image are generated as 
follows: 

(A) To interpolate the value of a red fill-in pixel at the 
pasition of a green pixel G, the non-zero coefficient 65 
values of the horizontal midpoint PFF shown in FIG. 
6A or the vertical midpoint PFF shown in FIG. 6B are 



12 



(8) 5 
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This averaged center midpoint PFF filter generates truer 
contour curves than the other two presented above, but 
requires more computation (because it has twelve non-zero 
coefficients) and produces edges that are less sharp. 

More generally, for every PFF described in this document 
except those for generating pixel fill-in values near the edge 
or corner of an image data array, the invention also includes 
versions in which the coefficients have been reversed with 
respect to either the X or Y axis, and in some cases the 
invention also includes a PFF whose coefficients are equal to 
the average of the described filter coefficients with the 
coefficients reversed with respect to one axis. 

One- and Two-Dimensional "Edge" and "Corner" 

Midpoint Pixel Filling Filters 25 

FIG. 6D depicts a pixel filling filter for a horizontal 
midpoint that is near an edge of the image data array. Using 
the convention that the fill-in pixel is located at position 0,0, 
the non-zero coefficient values for the edge-condition hori- 30 
zontal midpoint PFF are: 



used. The non-zero coefficient values of the horizontal 
midpoint PFF are used when the red pixels R closest to 
the red fill-in pixel have the same X coordinate as the 
red fill-in pixel. The non-zero coefficient values of the 
vertical midpoint PFF are used when the red pixels R 
closest to the red fill-in pixel have the same Y coordi- 
nate as the red fill-in pixel. The non-zero coefficient 
values are applied to a set of corresponding red pixels 
R adjacent to the red fill-in pixel in the image. The set 
of corresponding red pixels R are those red pixels R in 
the image that have the same position relative to the 
fill-in pixel as the non-zero coefficient value data 
points. 

(B) To interpolate the value of a red fill-in pixel at the 
position of a blue pixel B, the non-zero coefficient 
values of the center midpoint PFF shown in FIG. 6C are 
used. The non-zero coefficient values are applied to a 
set of corresponding red pixels R adjacent to the red 
fill-in pixel in the image. The set of corresponding red 
pixels R are those red pixels R in the image that have 
the same position relative to the fill-in pixel as the 
non-zero coefficient value data points. 

(C) To interpolate the value of a green fill-in pixel at the 
position of either a red pixel R or a blue pixel B, the 
non-zero coefficient values of the center midpoint PFF 
shown in FIG. 6C are used. However, in this case, the 
midpoint PFF is rotated 45° clockwise with respect to 
the orientation shown in FIG. 6C, resulting the follow- 
ing set of PFF coefficients: 



(12) 
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P-I.Z. 


P0,2, 


Pi. 2 > 




' -1 


0 




P-u* 


P0A< 


Pl.l 




0 


8 


0 


P-1,0, 


Po.Oi 


P1.0 


1 

" 16 


2 


0 


2 


P-1.-1 


P0.-1* 


Pu-i 




0 


8 


0 


P-l.2, 


P0.-2 


Px.-i > 




.-1 


0 


-1 , 



Since only green pixels are used in the PFF processing, 
and they are positioned in a checkerboard palter, the filter 
coefficients are also arranged in a checkerboard pattern. As 
usual, all filter coefficients not shown above are equal to 0. 
In an alternate embodiment, the filter coefficients shown in 
Equation 12 can be rotated 90°. 

The non-zero coefficient values are applied to a set of 
corresponding green pixels G adjacent to the green fill-in 
pixel in the image. The set of corresponding green pixels G 
are those green pixels G in the image that have the same 
position relative to the fill-in pixel as the non-zero coefficient 
value data points. 

(D) To interpolate the value of a blue fill-in pixel at the 
position of a red pixel R, the non-zero coefficient values 
of the center midpoint PFF shown in FIG. 6C are used. 
The non-zero coefficient values are applied to a set of 
corresponding blue pixels B adjacent to the blue fill-in 
pixel in the image. The set of corresponding blue pixels 
B are those blue pixels B in the image that have the 
same position relative to the fill-in pixel as the non-zero 
coefficient value data points, 
(F) To interpolate the value of a blue fill-in pixel at the 
position of a green pixel G, the non-zero coefficient 
values of the horizontal midpoint PFF shown in FIG. 
6A or the vertical midpoint PFF shown in FIG. 6B are 
used. The non-zero coefficient values of the horizontal 
midpoint PFF are used when the blue pixels B closest 
to the blue fill-in pixel have the same X coordinate as 
the blue fill-in pixel. The non-zero coefficient values of 
the vertical midpoint PFF are used when the blue pixels 



06/07/2004, EAST Version: 1.4.1 



US 6,496,608 Bl 



13 



14 



B closest to the blue fill-in pixel have the same Y 
coordinate as the blue fill-in pixel. The non-zero coef- 
ficient values are applied to a set of corresponding blue 
pixels B adjacent to the blue fill-in pixel in the image. 
The set of corresponding blue pixels B are those blue 
pixels B in the image that have the same position 
relative to the fill-in pixel as the non-zero coefficient 
value data points. 
In the digital camera embodiment described above, the 



the non-zero coefficient values for this horizontal quarter- 
point PFF are: 



(G-3.0. G-1.0. Gl.O- 03.0 



G 5 .o)=yg(1.4 t 12. 0,-1) 



(16) 



All other coefficient values for the horizontal quarter- 
point PFF are equal to zero. 

FIG. 8B depicts a horizontal quarter-point pixel filling 



interpolation of the red, green and blue fill-in pixel values is 10 fij ter f or ^ w hen the closest original data point is to the 



performed by the data processing circuitry 106 of the digital 
camera system 100. In the computer system embodiment 
mentioned earlier, the fill-in pixel interpolation is performed 
by the image processing module 178 of the general-purpose 
computer system 150. 

Quarter-Point Pixel Filling Filters 

A more general filter equation for PFF than equation 3 
above is: 



fi,t = J] Pk-2ruJ-2w d iJ 



(13) 



where n is a positive integer (with typical values being 1, 2 
and 4), and ? kl are the coefficients of the filter. This 
generalized equation basically indicates that the fill in pixels 
are based on a set of evenly spaced (at least with respect to 
the index values) original pixels. 

Referring to FIG. 7A there is shown a pixel filling filter for 
computing fill-in pixel values for an image that has been 
magnified by a factor of sixteen. The general pixel filling 
formula for generating the fill-in pixel values is as follows: 



(14) 



8r,s - 2 Qi^2t.s-7lfkJ 



right of the quarter-point whose value is being determined. 
The non-zero coefficient values for this horizontal quarter- 
point PFF are: 
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The digital image represented by the g FfS values has the 
same properties as those discussed above with respect to the 
midpoint pixel filling filter. 

FIG, 7B shows a two-level pixel filling filter for accom- 
plishing the same level of magnification. In this second 
embodiment, fill-in pixel values f* , are first computed for all 
the midpoints in accordance with the midpoint PFF's 
described above. Those fill-in pixel values are then available 
for compulation of the quarter-pointer fill-in pixels g^ in 
accordance with a second level pixel filling filter equation: 



In this second implementation, the second level filter 
could, potentially, use PFF's identical to those used in the 
first level filter. However, in the preferred embodiment, the 
second level filter uses a different set of PFF coefficients 
Q r ^, because re -use of the first level filter coefficients causes 
the resulting picture to lose its sharpness. The coefficients 
of the second level filter are selected to preserve sharp 
edges in the image being magnified. 

Therefore, our attention will now focus on the embodi- 
ment shown in FIG. 7B. 

FIG. 8A depicts a horizontal quarter-point pixel filling 
filler for use when the closest original data point is to the left 
of the quarter-point whose value is being determined. Using 
the convention that the fill-in pixel is located at position 0,0, 



(C?-5,o. G-3.0, G 



i.o. Gi,o. Q3,o ) = 



1,0, 12,4, 1) 



(17) 



For every quarter-point fill-in pixel value shown in FIGS. 
20 8A and 8B, except those closest to the edges of the image 
array, the fill-in value is determined by summing four of the 
five closest data points multiplied by the coefficients shown 
in the figures. 

FIG. 8C depicts a vertical quarter-point pixel filling filter 
for use when the closest original data point is below the 
quarter-point whose value is being determined. The non- 
zero coefficient values for this vertical quarter-point PFF are: 



(Go.-3.Go,-i.Go,i.Go,3,Go.s) 



:-(l,4, 12, 0. -1) 



(18) 



FIG. 8D depicts a vertical quarter-point pixel filling filter 
for use when the closest original data point is above the 
quarter-point whose value is being determined. The non- 
35 zero coefficient values for this vertical quarter-point PFF are: 



(Go.-5.Go.-3,Go.-l,Go.l.Go.3> = 12 ' 4 ' 1) 



(19) 



FIGS. 8E, 8F and 8G depict three different two- 
dimensional center quarter-point pixel filling filters for three 
different classes of center quarter-point pixels. When the 
quarter-point whose value is being generated is located on a 
45 diagonal line between one original data point and one 
midpoint as shown in FIG. 8E, the non-zero PFF coefficients 
are: 



(15) 50 



G-3.5, G-l.5. 01,5, G3.5* GS.5 

G-3.3, G-l.3. Gl.3. G3.3- G5.3 

G-3.1, G-1,1. Gu, G3.1. G5.1 

G-3.-1. G-i.-j, G1.-1. G 3 .-i, Gs,-i 

G -3.-3. G-1.-3. Gl,-3, G3.-3, G5.-3 



0 0-10 0 

0 0 0 0 0 

1 0 12 0 -1 
0 4 0 0 0 
0 0 10 0 



(20) 



55 



When the quarter-point whose value is being generated is 
located on a horizontal line between two midpoints, but is 
not horizontally colinear with any original data points, as 
shown in FIG. 8F, the non-zero PFF coefficients are: 



60 



65 



G-3.6, 


G-L.6, 


G1.6, 


G3.6. 


Gs.6 


G-3.4, 


G-..4, 


G1.4. 


G3.4, 


G5.4 


G-3,2, 


G-l.2. 


G1.2, 


G3.2, 


Qsj 


G-3.0, 


G-1.0, 


G1.0. 


G3.0, 


G5.0 


G-3.-2, 


G-1.-2. 


G1.-2, 


G3.-2. 


G5.-2 



(21) 
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When the quarter-point whose value is being generated is 
located in the middle between one original data point and 
three midpoints that together form a square (with the 
quarter-point in the middle of the square), as shown in FIG. 
8G, the non-zero PFF coefficients are: 



G-3,3. e-u. Gu> fe. 

C-3,1. G-i.i. 0u, Qxu 

Q-3,-1, {?_!,_!, 03.-1, 05.-1 

0,3,-3, 0-1.-3, 01.-3, 03.-3, 05,-3 

0-3.-5, 0-1.-5. 01.-S, 03.-5, 05.-5 



(22) 



15 



( 0 0-10 0 

0 8 8 0 0 

-1 8 12 0 -1 

0 0 0 0 0 

0 0-100 



It is noted that FIG. 8G and 8E are different. The PFF for 
FIG. 8G is not the same as the PFF for FIG. 8E because the 
PFF is generally symmetric only for fill-in points that are 
symmetric with respect to axes parallel to the x=y diagonal 
and is not symmetric with respect to fill-in points that are 
symmetric with respect to either the horizontal or vertical 
axes. 
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(7 0 ,. 7 , 7 0 ,.3, To,.,, To, 5 J = ^(-1, 13, 57, -5) 



(25) 



When the closest original data point is above the quarter- 
point whose value is being determined, as shown in FIG. 8D, 
the non-zero coefficient values for the vertical quarter-point 
PFF are: 



10 



tfa-s, 7a- 1, Tq.3, 7b, 7 ) = ^(-5, 57, 13, - 



(26) 



When the quarter-point whose value is being generated is 
located on a diagonal line between two original data points, 
as shown in FIG. 8E, the non-zero PFF coefficients for the 
center quarter-point PFF are: 



20 



25 



7-7.1. ^-3.1, 7*1,1. 7 51 
7-7,-3. T- 3-3, 7i-3, r 3 _3 
7-7.-7. 7-3.-7, 7J.-7, 7 5 .-7 



1 

728 



( 0 -1-8 0 

2 21 100 -8 

1 10 21 -1 
10-1-20 



(27) 
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When the quarter-point whose value is being generated is 
located on a horizontal line between two midpoints, but is 
not horizontally colinear with any original data points, as 
shown in FIG. 8F, the non-zero PFF coefficients for the 
center quarter-point PFF are: 



' 7-7,6^ 7-3,6, 7 it 6, 7 5 ,6 

7-7.2, 7_ 3i2 , 7 lt2 , 7 5t2 

7-7.-2, 7-3,-2, 71.-2, 73,-2 

7_ 7> _$, 7_3,_6, ^1.-6. 7*s -6 



1 

128 



0-1-7 0 

-2 17 61 -4 

-2 17 61 -4 

0 -1-7 0 ) 



(28) 



Quarter-Point PFF Using Only Original Data Points 

As indicated above in the discussion of FIG. 7A, the 
quarter-point fill-in pixel values can be computed solely 4 o 
from the original data points, using Equation 14. The fol- 
lowing are the pixel filling coefficients for Equation 14, for 
each of the quarter-point positions for the fill-in pixel. 

When the closest original data point is to the left of the 
quarter-point whose value is being determined, as shown in 45 
FIG. 8A, the non-zero coefficient values for the horizontal 
quarter-point PFF are: 



(7-7.o, 7-3.o,7- l(0 ,7 s .o)= -(-1. 13.57, - 



■5) 



When the quarter-point whose value is being generated is 
located in the middle between one original data point and 
three midpoints that together form a square (with the 
quarter-point in the middle of the square), as shown in FIG. 
8G, the non-zero PFF coefficients for the center quarter- 
point PFF are: 



(23) 



( 7.7.7 . 7-3.7, 7,,7. 7 5 .7 

7-7.3, 7-3,3, 7l.3. 7 5 , 3 

7_ 7 ,_ i . 7-3.-1, 71-,, 7 5 ,_, 

7-7-5, 7_ 3i - S , 7|_ 5 , 7 5 - 5 



1 

128 



r 0 -1-2 0 

-1 10 21 -1 

-2 21 100 -8 

I 0 -1 -8 0 J 



(29) 
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As before, the coefficient values use the convention that 
the fill-in pixel is located at position 0,0. 

When the closest original data point is to the right of the 
quarter-point whose value is being determined, as shown in 
FIG. 8B, the non-zero coefficient values for the horizontal 
quarter-point PFF are: 



(7-5.0, 7-ko. 73.0. 7*7.0,) = g|(-5. 57, 13, - 



When the closest original data point is below the quarter- 
point whose value is being determined, as shown in FIG. 8C, 
the non-zero coefficient values for the vertical quarter-point 
PFF are: 



Pixel Filling when Image Magnification is not an 
Integer Power of Four 

To magnify an image by a specified factor other than an 
55 integer power of four, the image is magnified in two stages. 
In the first stage the image is magnified by a factor that is 
smaller than the specified factor, but which is an integer 
power of four. (The first stage is skipped if the specified 
magnification factor is less than four.) The newly created 
(24) 60 pixels are generated using the pixel filling filter of the 
present invention. In the second stage the pixel values 
resulting from the first stage are remapped to new locations 
based on the specified magnification factor. This mapping 
will usually map some pixel values to non-pixel positions, 
65 and will always leave many pixel positions with no assigned 
pixel value. Anew type of filter, herein called a "Barycentric 
interpolation filter," is used to compute pixel values at each 
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of the pixel positions P not assigned a pixel value by the 
remapping. For each such pixel position P, a reference 
triangle is selected by selecting the three closest pixel 
values, at positions A, B and C, having pixel values a, b and 
c, respectively. The coordinates of those pixel values and the 5 
pixel position P for which a pixel value p is to be generated 
are mapped into the Barycentric coordinate system. The 
coordinates (u, v, w) of the pixel position P in the Barycen- 
tric coordinate system are then used as weights for gener- 
ating a pixel value p=*ua+vb+wc that lies on the plane that 30 
passes through the vertices A, B and C of the reference 
triangle. 

More specifically, FIG. 9 is a flow chart of the method of 
generating pixel data values when an image is magnified by 
a factor that is not equal to an integer power of four. In step ]S 
201, determine whether the magnification power is an inte- 
ger power of four. If not, in step 203, select the largest power 
of four that is smaller than the magnification power. In step 
204, remap the original pixels d* , to locations based on the 
selected magnification power of four, as described earlier; in 20 
other words, magnify the image by the selected magnifica- 
tion power of four. In step 205, generate a fill-in pixel value 
i kJ at each pixel location with the pixel filling method of 
Equations 3, 14 and/or 15, as described earlier. In step 206, 
remap the pixels to locations based on the actual magnifi- 25 
cation power; in other words, magnify the image by the 
actual magnification power. In step 207, generate a fill-in 
pixel value at each pixel location with a Barycentric inter- 
polation method using the remapped pixels. The fill-in pixel 
values are used to form the magnified image. The Barycen- 30 
trie interpolation method performs the interpolation by 
applying Barycentric coordinate weights, as described in the 
next section of this document. The Barycentric interpolation 
method comprises two steps: (1) select a reference triangle 
having vertices at three remapped pixels at the actual 35 
magnification power (from step 206) closest to the pixel 
location P and (2) generate the fill-in pixel value p at the 
pixel location with Barycentric interpolation using the three 
closest remapped pixels. 

In an alternate embodiment, linear spline interpolation 
can be used in place of Barycentric interpolation. 

Barycentric Interpolation 

Barycentric interpolation as applied to the problem of 45 
image magnification is explained next. 

FIG. 10 depicts the pixels generated by the method of 
FIG. 9 when an image is magnified by a factor of nine (i.e., 
three times in each dimension). Initially the image is mag- 
nified by a factor of four and the fill in pixel values at that so 
magnification are generated using the techniques described 
above. Those pixels are then remapped to positions at a 9x 
magnification, resulting in some pixel values being mapped 
to integer pixel positions and some being mapped to non- 
integer positions. The square boxes with x's in them repre- 55 
sent pixel values mapped to integer pixel positions, the 
empty square boxes represent pixel values mapped to non- 
integer pixel positions, and the x's represent integer pixel 
positions to which no value in initially assigned when the 4x 
pixels are initially mapped to their 9x magnification posi- 60 
tions. As can be seen, some of the unassigned integer pixel 
positions are either horizontally colinear or vertically colin- 
ear with remapped pixel values, and those pixel positions are 
assigned values using simple linear interpolation. In other 
words, in this particular situation, Barycentric interpolation 65 
is identical to simple linear interpolation. Pixel positions that 
are not horizontally or vertically colinear with remapped 



40 



pixel values are assigned values using the Barycentric 
interpolation formula: 

xna+vtn-wc 

where u+v+w=l and pixel values a, b and c represent the 
three pixels closest to the pixel position for which an 
interpolated value is being generated. The three closest 
pixels will, in general, form a triangle with the "selected" 
pixel (i.e., the one for which an interpolated pixel value is 
being generated) falling in the interior of the triangle. In this 
particular case, u=v=w=l/3. More generally, at other 
magnifications, (u,v,w) are generated in accordance with the 
equations: 

(*2?3 - ^2) + toy - *y 3 ) + (xy 2 - 





■jcj^J + teyi 


-xiyi) + (xiy 2 


-X2yi) 




- x y y) + {x 3 yi 


-x l y^) + (x 1 y- 


-xyi) 


tew - 


*3_y2) + toyi 


-xiy 3 ) + {xiy z 


-x 2 yi) 


toy 


-xy 2 ) + (xy l ■ 


-xiy) + (xiy 2 - 


x 2 yi) 


tew - 






-x 2 yO 



where (x^yj, fey^ and (x 3 ,y 3 ) are the locations of the 
three pixels whose position and values define the plane (in 
three dimensional space) on which the location (x,y) is 
supposed to be located. It is noted that the denominators of 
the equations for u, v and w are identical, and that the 
numerators are linear with respect to both x and y. 

RGB Data Compression 

After performing interpolation as described above, to 
generate the red, green and blue values not generated by the 
image sensor, the resulting image data can be compressed by 
converting it to YUV (or YC fc C r ) data and then applying 
standard luminance/chrominance compression techniques to 
generate a compressed image file. When the file is 
decompressed, the resulting decompressed YUV is then 
reconverted back to RGB format. 

In a second embodiment, the pixels data generated by the 
image sensor is divided into even and odd rows. 
Interpolation, as described above, is performed to generate 
the red, green and blue values not generated by the image 
sensor. In this embodiment, after the RGB values are con- 
verted to YUV (or YC 6 C r ) data, the pixel data for the even 
rows is compressed separately from the pixel data for the 
odd rows, and the two sets of compressed data are stored 
together as an image file. When the file is decompressed, the 
resulting decompressed YUV for the odd and even rows are 
merged, and the resulting matrix of values is then recon- 
verted back to RGB format. 

While the present invention has been described with 
reference to a few specific embodiments, the description is 
illustrative of the invention and is not to be construed as 
limiting the invention. Various modifications may occur to 
those skilled in the art without departing from the true spirit 
and scope of the invention as defined by the appended 
claims. 

What is claimed is: 

1. A method of generating interpolated pixel data values, 
comprising: 

providing an initial set of image data, comprising a subset 
of data values in a data array i kl representing data 
values at corresponding positions (k,l); such that some 
of the i kl values are initially undefined; 
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generating an interpolated value for each of a plurality of 
initially undefined f kt values in accordance with the 
filter equation: 



10 



15 



20 



25 



wherein 

n is a positive integer; 

d £j - are the subset of f kl values which have defined 
values prior to the generation of the interpolated f k>1 
values, such that f^^-d^ ^ or mte S er values of i 
and j that fall within the initial set of image data; 

P A ; are coefficients, no more than eighteen of which 
have non-zero values; 

each non-zero P kl coefficient is an integer divided by an 
integer power of two; and 

the set of interpolated i k l values in conjunction with the 
d (J values represent a smooth surface that is 
continuous, and that has a two-dimensional spatial 
first derivative that is continuous. 

2. The method of claim 1, wherein the smooth surface 
represented by the set of interpolated f k j values in conjunc- 
tion with the d f - . values has at each of a plurality of locations 
(k,l) for which an interpolated value is generated a tangent 
plane that is substantially parallel to a plane formed by 
adjacent ones of the data points (k,l) corresponding to data 
values dy. 

3. The method of claim 1, wherein 
whenever a local contiguous set of the initial data values 

d lV all fall on a polynomial surface, the interpolated f kJ 
values that are located between the initial data points 
also fall on that same polynomial surface. 

4. The method of claim 1, wherein the non-zero 35 
coefficients, when summed, have an associated sum equal to 

1. 

5. An image processing apparatus for generating interpo- 
lated pixel data values, comprising: 

an image memory for storing an initial set of image data, 40 
comprising a subset of data values in a data array f kJ 
representing data values at corresponding positions 
(k,l); such that some of the f*, values are initially 
undefined; and 

data processing circuitry coupled to the image memory 
for generating an interpolated value for each of a 
plurality of initially undefined f* , values in accordance 
with the filter equation: 



6. The image processing apparatus of claim 5, wherein the 
smooth surface represented by the set of interpolated t kl 
values in conjunction with the 6 tJ values has at each of a 
plurality of locations (k,l) for which an interpolated value is 
generated a tangent plane that is substantially parallel to a 
plane formed by adjacent ones of the data points (k,l) 
corresponding to data values d iV . 

7. The image processing apparatus of claim 5, wherein 
whenever a local contiguous set of the initial data values 

d lV the interpolated i kJ values that are located between 
the initial data points also fall on that same polynomial 
surface. 

8. The image processing apparatus of claim 5, wherein the 
non-zero P k J coefficients, when summed, have an associated 
sum equal to 1. 

9. The image processing apparatus of claim 5, wherein 
the data processing circuitry applies a two-dimensional 

center midpoint pixel filling filter to generate a data 
value f k j at a position (k,l) that is located at a center 
position in the data array between four positions of the 
data array which have defined values, but which is not 
at the same k or 1 position in the data array as any of 
those four positions; 
the only non-zero coefficients in the applied pixel 
filling filter are selected from the set consisting of: 
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wherein 

n is a positive integer; 55 

d (V are the subset of f M values which have defined 
values prior to the generation of the interpolated 
values, such that f^^d^- for integer values of i 
and j that fall within the initial set of image data; 

? kl are coefficients, no more than eighteen of which 60 
have non-zero values; 

each non-zero P k l coefficient is an integer divided by an 
integer power of two; and 

the set of interpolated f* , values in conjunction with the 
d (J - values represent a smooth surface that is 65 
continuous, and that has a two-dimensional spatial 
first derivative that is continuous. 
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and 



and 



10. The image processing apparatus of claim 5, wherein 
the data processing circuitry applies a two-dimensional 
center midpoint pixel filling filter to generate a data 
value f w at a position (k,l) that is located at a center 
position in the data array between four positions of the 
data array which have defined values, but which is not 
at the same k or 1 position in the data array as any of 
those four positions; 
when, the position (k,l) is near a left edge of the data array, 
the only non-zero coefficients P kl in the applied pixel 
filling filter, are: 
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11. The image processing apparatus of claim 5, wherein 
the data processing circuitry applies a two-dimensional 
center midpoint pixel filling filter to generate a data 
value f k j at a position (k,l) that is located at a center 
position in the data array between four positions of the 
data array which have defined values, but which is not 
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at the same k or 1 position in the data array as any of 
those four positions; 
when the position (k,l) is near a top left corner of the 
image data array, the only non-zero coefficients P^ j in 
the applied pixel filling filter, are: 



{ ^-3.3. 
^-3.1- 
P-3.-1 . 



^-1.1 , 



Pl.3. 



/>3,3 
P3.-3 
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12. The image processing apparatus of claim 5, wherein 
the initial set of image data includes red, green and blue 

pixel values arranged in the data array in a Bayer 

mosaic pattern; 
the data processing circuitry applies a two-dimensional 

center midpoint pixel filling filter to generate a data 

value t kJ at a position (k,l) representing a green pixel 

value at a position in the data array initially assigned 

only a blue or red pixel value; and 
the only non-zero coefficients P w in the applied pixel 

filling filter, are: 
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13. The image processing apparatus of claim 5, wherein 

the data processing circuitry first applies one or more 
midpoint pixel filling filters to generate a set of mid- 
point interpolated data values located at midpoints 
between the positions in the data array having data 
values from the initial set of image data; 

the data processing circuitry next applies a two- 
dimensional center quarter-point pixel filling filter to 
generate a data value i kl at a position (k,l) that is 
located on a diagonal line between a first position of the 
data array having a defined value from the initial set of 
image data and a second position located at a midpoint 
of a diagonal line segment between the first position 
and a third position of the data array having a defined 
value from the initial set of image data; 

the only non-zero PFF coefficients in the applied quarter- 
point pixel filling filter are: 
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14. The image processing apparatus of claim 5, wherein 
the data processing circuitry first applies one or more 
midpoint pixel filling filters to generate a set of mid- 
point interpolated data values located at midpoints 
between the positions in the data array having data 
values from the initial set of image data; 
the data processing circuitry next applies a two- 
dimensional center quarter-point pixel filling filter to 
generate a data value i kl at a position (k,l) that is 
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located on a horizontal line between two of the mid- 
points in the data array, and that it is not horizontally 
colinear with any of the positions in the having data 
values from the initial set of image data; 
the only non-zero PFF coefficients in the applied quarter- 
point pixel filling filter are: 



G-3.6, 

G-3.4, 

G-3.2, 
G-3.0, 
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15. The image processing apparatus of claim 5, wherein 

the data processing circuitry first applies one or more 
midpoint pixel filling filters to generate a set of mid- 
point interpolated data values located at midpoints 
between the positions in the data array having data 
values from the initial set of image data; 

the data processing circuitry next applies a two- 
dimensional center quarter-point pixel filling filter to 
generate a data value f M at a quarter-point position (k,l) 
that is located in the middle of a square formed by one 
of the positions having a data value from the initial set 
of image data and three of the midpoints; 

the only non-zero PFF coefficients in the applied quarter- 
point pixel filling filter are: 



( 0-3.3, 
0-3.1, 
0-3.-1, 

G-3.-3. 
G-3.-5, 



G-u, 
G-u, 
G-i.-i, 

0-1.-3, 



01.3, 

Gu. 
G1.-1. 

Gl,-3, 



0^,-5, Gl,-5- 



03.3, 
(23.1. 

03.-1. 

03.-3. 
03.-5. 



05.3 
05.1 
05.-1 
05.-3 
G5.-5 



0 -1 



16. The image processing apparatus of claim 5, wherein 

the data array includes original data positions having data 
values from the initial set of image data, midpoint 
positions located at midpoints between the original data 
positions, and quarter-point positions located at mid- 
points between the positions defined by the original 
data positions and midpoint positions; 

the data processing circuitry next applies a two- 
dimensional center quarter-point pixel filling filter to 
generate a data value f*, at a position (k,l) that is 
located on a diagonal line between a first position of the 
data array having a defined value from the initial set of 
image data and a second position located at a midpoint 
of a diagonal line segment between the first position 
and a third position of the data array having a defined 
value from the initial set of image data; 

the only non-zero PFF coefficients in the applied quarter- 
point pixel filling filter are: 



( r-7,5, 

7--7.1. 
T-7,-3. 
7*-7.-7. 



7"-3 t1 , 
T-3.-J, 
7"-3.-7. 



7-1.5, 
7*1.1. 

7*1.-3 , 

7-1.-7. 



75.5 

7*5.1 

7-5.-3 

7*5.-7 



1 

128 



100 
21 
-2 



17. The image processing apparatus of claim 5, wherein 
the data array includes original data positions having data 
values from the initial set of image data, midpoint 
positions located at midpoints between the original data 
positions, and quarter-point positions located at mid- 



06/07/2004, EAST version: 1.4.1 



US 6,496,608 Bl 



23 



points between the positions defined by the original 
data positions and midpoint positions; 

the data processing circuitry next applies a two- 
dimensional center quarter-point pixel filling filter to 
generate a data value ( kt at a position (k,l) that is 
located on a horizontal line between two of the mid- 
points in the data array, and that it is not horizontally 
colinear with any of the positions in the having data 
values from the initial set of image data; 

the only non-zero PFF coefficients in the applied quarter- 
point pixel filling filter are: 



7.6. 



7*-7.-; 
7--7.-I 



r-3.2, 

7*-3.-2, 
T-3,-6, 



7* 1( <s, 

7*1.2. 

7l-«, 



Ts.6 

7*5.3 
7*5.-2 

r 5l - 6 



128 



0 
-2 
-2 

0 



0 
-4 
-4 

0) 



18. The image processing apparatus of claim 5, wherein 

the data array includes original data positions having data 
values from the initial set of image data, midpoint 
positions located at midpoints between the original data 
positions, and quarter-point positions located at mid- 
points between the positions defined by the original 
data positions and midpoint positions; 

the data processing circuitry applies a two-dimensional 
center quarter-point pixel filling filter to generate a data 
value i k { at a quarter-point position (k,l) that is located 
in the middle of a square formed by one of the positions 
having a data value from the initial set of image data 
and three of the midpoints; 

the only non-zero PFF coefficients in the applied quarter- 
point pixel filling filter are: 



r.7.7. 



r-3.7 

7*-33 
7-3.-1 



7*1,7 > 
7*1.-5. 



7*5.7 
7*53 

7*5.-5 



128 



-2 
21 
100 



19. The image processing apparatus of claim 5, wherein 
the initial set of image data includes values for alternating 

horizontal rows of the data array; 
the data processing circuitry generates values for a hori- 
zontal row of the data array that does not have values 
from the initial set of image data by applying a vertical 
midpoint pixel filling filter in accordance with the filter 
equation; and 

the only non-zero coefficient values for the vertical mid- 
point pixel filling filter are: 



1 



-(-1,9, 9,-1). 



20. A general-purpose computing apparatus for generating 
interpolated pixel data values, comprising: 
a data processing unit; and 

a memory coupled to the data processing unit, including: 
an image storage area for storing an initial set of image 
data, comprising a subset of data values in a data 
array f kl representing data values at corresponding 
positions (k,l); such that some of the f*, values are 
initially undefined; and 
an image processing module for generating, in con- 
junction with the data processing unit, interpolated 



24 



values for initially undefined i kJ values in accor- 
dance with the filter equation: 

fk.l ~ Yj P k-2ni,t-2iydi.j 



wherein 

n is a positive integer; 
!0 d, v are the subset of i kl values which have defined 

values prior to the generation of the interpolated 
i k l values, such that ^ni^j^i^ f° r integer values 
of i and j that fall within the initial set of image 
data; 

15 P k j are coefficients, no more than eighteen of which 

have non-zero values; 
each non-zero V kI coefficient is an integer divided by 

an integer power of two; and 
the set of interpolated f k j values in conjunction with 
20 the d, v values represent a smooth surface that is 

continuous, and that has a two-dimensional spatial 
first derivative that is continuous. 
21. The general-purpose computing apparatus of claim 
20, wherein the smooth surface represented by the set of 
25 interpolated f kJ values in conjunction with the d^ values has 
at each of a plurality of locations (k,l) for which an inter- 
polated value is generated a tangent plane that is substan- 
tially parallel to a tangent plane formed by adjacent ones of 
the data points (k,l) corresponding to data values d, ,. 
30 22. The general-purpose computing apparatus of claim 
20, wherein 

whenever a local contiguous set of the initial data values 
d tJ . all fall on a polynomial surface, the interpolated i kl 
values that are located between the initial data points 
35 also fall on that same polynomial surface. 

23. The general-purpose computing apparatus of claim 
20, wherein the non-zero P^, coefficients, when summed, 
have an associated sum equal to 1. 

24. A computer program product for generating interpo- 
40 lated pixel data values, for use in conjunction with a 

computer system, the computer program product comprising 
a computer readable storage medium and a computer pro- 
gram mechanism embedded therein, the computer program 
mechanism comprising: 
45 an image module including instructions for providing an 
initial set of image data, comprising a subset of data 
values in a data array i kl representing data values at 
corresponding positions (k,l), such that some of the i k l 
values are initially undefined; and 
50 a pixel filling filter module including instructions for 
generating an interpolated value for initially undefined 
f k l values in accordance with the filter equation: 



55 



60 



65 



wherein 

n is a positive integer; 

d )V are the subset of f*, values which have defined 
values prior to the generation of the interpolated f k l 
values, such that f 2nJ) 2« / ss d ll/ for integer values of i 
and j that fall within the initial set of image data; 

P* ; are coefficients, no more than twelve of which have 
non-zero values; 

each non-zero P^ , coefficient is an integer divided by an 
integer power of two; and 
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the set of interpolated t kl values in conjunction with the 
dy values represent a smooth surface that is 
continuous, and that has a two-dimensional spatial 
first derivative that is continuous. 

25. The computer program product of claim 24, wherein 
the smooth surface represented by the set of interpolated f^ 
values in conjunction with the d (J values has at each of a 
plurality of locations (k,l) for which an interpolated value is 
generated a tangent plane that is substantially parallel to a 
tangent plane formed by adjacent ones of the data points 
(k,l) corresponding to data values d tJ -. 

26. The computer program product of claim 24, wherein 
whenever a local contiguous set of the initial data values 

t kh for k and 1 both even numbers, all fall on a 
polynomial surface, the interpolated f kI values that are 
located between the initial data points also fall on that 
same polynomial surface. 

27. The computer program product of claim 24, wherein 
the non-zero coefficients, when summed, have an asso- 
ciated sum equal to 1. 

28. A method of generating interpolated pixel data values, 
comprising: 

providing an initial set of image data, comprising a subset 
of data values in a data array g rs representing data 
values at corresponding positions (r,s); such that some 
of the g r ^ values are initially undefined; 

generating an interpolated value for each of a plurality of 
initially undefined g rA values in accordance with the 
filter equation: 

wherein 

d (J are the subset of g rA values which have defined 
values prior to the generation of the interpolated g r ^ 
values, such that g^^d^- for integer values of i and 
j that fall within the initial set of image data; 

T r _ 4l> _ 4 y are coefficients, no more than sixteen of which 
have non-zero values; 

each non-zero T r _ 4l> _ 4; coefficient is an integer divided 
by an integer power of two; and 
the set of interpolated g r ^ values in conjunction with the 

d iV values represent a smooth surface that is 

continuous, and that has a two-dimensional spatial first 

derivative that is continuous. 

29. A method of generating interpolated pixel data values, 
comprising: 

providing an initial set of image data, comprising a subset 
of data values in a data array i kJ representing data 
values at corresponding positions (k,l), such that some 
of the i k j values are initially undefined; 

generating an interpolated value for each of a plurality of 
initially undefined f kl values in accordance with the 
filter equation: 

fkj = ^Pk-2i,t-2jdiJ 

wherein 

d^- are the subset of f k , values which have defined 
values prior to the generation of the interpolated i kJ 
values, such that ^iaj^'j ^ or mte S er values of i and 
j that fall within the initial set of image data; 
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P*-2/,/-2/ are coefficients, no more than twelve of which 
have non-zero values; and 

each non-zero P k -2u-2j coefficient is an integer divided 
by an integer power of two; 
5 providing a data array g rA representing data values at 
corresponding positions (r,s) including the data val- 
ues in the data array f kt such that g 2 k f zt ,a h t i f° r 
integer values of k and 1 that fall within the data array 
i kJ and such that the other g rA values are initially 
3Q undefined; and 

generating an interpolated value for each of a plurality 
of initially undefined g r ^ values in accordance with 
the filter equation: 

k.l 

wherein 

Qr-2*,*-2/ are coefficients, no more than eight of 
20 which have non-zero values; and 

each non-zero Q r _2* )JF _2/ coefficient is an integer 
divided by an integer power of two. 
30. A method of generating interpolated pixel data values 
for an image when magnifying the image by a magnification 
25 factor that is not equal to an integer power of four, com- 
prising: 

performing a first stage magnification of the image, fol- 
lowed by a second stage magnification of the image; 
the step of performing the first stage magnification of the 
30 image including: 

selecting a largest power of four, n, that is smaller than 

the magnification factor; 
providing an initial set of image data, comprising a 
subset of data values in a data array h, representing 
35 data values at corresponding positions (t,u) in the 

image magnified by a factor of n; such that some of 
the h, M values are initially undefined; and 
generating an interpolated value for each of a plurality 
of initially undefined h, u values in accordance with 
4 0 the FIR filter equation: 



wherein d JV - are the subset of h tu values which have 
defined values prior to the generation of the inter- 
polated h t u values, such that hy-^d^- for integer 
values of i and j that fall within the initial set of 
50 image data, and W, M are coefficients; and 

wherein the set of interpolated h, values in con- 
junction with the d fj - values represent a smooth 
surface that is continuous, and that has a two- 
dimensional spatial first derivative that is continu- 
55 ous; 

the step of performing the second stage magnification of 
the image including: 

remapping data values in the data array h,„ to new 
positions (t\u') based on the magnification factor; 
60 and 

generating an interpolated value at positions (t,u) in the 
image that did not receive a data value from the 
remapping step by interpolating, for each such 
position, at least one data value from at least one 
65 corresponding position (t\u'). 

31. The method of claim 30, wherein the generating step 
includes: 
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selecting three remapped data values at positions (t',u') 

closest to the position (t f u); and 
generating the interpolated value at the position (t,u) with 

Barycentric interpolation using the three closest 

remapped pixels. 

32. The method of claim 31, wherein the smooth surface 
represented by the set of interpolated h r u values in conjunc- 
tion with the d,- ■ values has at each of a plurality of locations 
(k,l) for which an interpolated value is generated a tangent 
plane that is parallel to a tangent plane formed by adjacent 
ones of the data points (k,l) corresponding to data values d lV . 

33. The method of claim 31, wherein 

whenever a local contiguous set of the initial data values 
h fK for k and 1 both even numbers, all fall on a 
polynomial surface, the interpolated h t u values that are 
located between the initial data points also fall on that 
same polynomial surface. 

34. The method of claim 31, wherein the non-zero W, w 
coefficients, when summed, have an associated sum equal to 
1. 

35. The method of claim 30, wherein 

no more than eighteen of the W /M coefficients have 

non-zero values; and 
each non-zero W, u coefficient is a equal to an integer 

divided by an integer power of two. 

36. A computer program product for generating interpo- 
lated pixel data values for an image when magnifying the 
image by a factor that is not equal to an integer power of 
four, for use in conjunction with a computer system, the 
computer program product comprising a computer readable 
storage medium and a computer program mechanism 
embedded therein, the computer program mechanism com- 
prising: 

an image module including instructions for providing an 
initial set of image data, comprising a subset of data 
values in a data array h t u representing data values at 
corresponding positions (t,u) in the image magnified by 
a factor of n; such that some of the b tu values are 
initially undefined; 

instructions for selecting a largest power of four, n, that is 
smaller than the magnification factor; 

a pixel filling filter module including instructions for 
generating an interpolated value for each of a plurality 
of initially undefined h, values in accordance with the 
FIR filter equation: 



wherein d, v are the subset of h 



values which have 
defined values prior to the generation of the inter- 
polated h tu values, such that h^^-d^- for integer 
values of i and j that fall within the initial set of 
image data, and W, M are coefficients; 



15 



20 



25 



30 



an image magnification module including instructions for: 
remapping data values in the data array h, u to new 
positions (t'.u') based on the magnification factor; 
and 

generating in the image an interpolated value for posi- 
tions (t,u) in the image that did not receive a data 
value from the remapping; 
the instructions for generating an interpolated value at 
position (t,u) including instructions for: 
selected three data values at remapped positions (t'.u 1 ) 

closest to the position (t,u); and 
generating the interpolated value at the position (t,u) 
with Barycentric interpolation using the three 
selected data values. 
37. An image processing apparatus for generating inter- 
polated pixel data values for an image when magnifying the 
image by a factor that is not equal to an integer power of 
four, for use in conjunction with a computer system, com- 
prising: 

an image memory for storing an initial set of image data, 
comprising a subset of data values in a data array i k [ 
representing data values at corresponding positions 
(k,l); such that some of the i kl values are initially 
undefined; and 

means for selecting a largest power of four, n, that is 
smaller than the magnification factor; 

a pixel filling filter for generating an interpolated value for 
each of a plurality of initially undefined h, u values in 
accordance with the FIR filter equation: 



.u-yfnj °<J 



wherein d ( j are the subset of h lfW 



45 



50 



values which have 
defined values prior to the generation of the inter- 
polated h t u values, such that hva^-^d/j for integer 
values of i and j that fall within the initial set of 
image data, and W, are coefficients; and 
an image magnification module that remaps data values in 
the data array h ru to new positions (t',u') based on the 
magnification factor, and generates an interpolated 
value at positions (t,u) in the image that did not receive 
a data value from the remapping; 
wherein the image magnification module generates an 
interpolated value for a position (t,u) by selecting three 
remapped data values at positions (t',u') closest to the 
position (t,u), and generating the interpolated value at 
the position (t,u) with Barycentric interpolation using 
the three selected remapped data values. 
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